{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2a916a6e-ffc8-4750-a90a-48add0b9a2af",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/pathwaycom/pathway-examples/blob/main/documentation/pathway_intro.ipynb\" target=\"_parent\"><img src=\"https://pathway.com/assets/colab-badge.svg\" alt=\"Run In Colab\" class=\"inline\"/></a><a href=\"https://github.com/pathwaycom/pathway-examples/blob/main/documentation/pathway_intro.ipynb\" target=\"_parent\"><img src=\"https://pathway.com/assets/gh-badge.svg\" alt=\"View in Github\" class=\"inline\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "notebook-instructions",
   "metadata": {
    "id": "notebook-instructions"
   },
   "source": [
    "# Introduction to programming with Pathway\n",
    "\n",
    "[Pathway](https://pathway.com/developers/user-guide/introduction/welcome) is a Python data processing framework for analytics and AI pipelines over data streams. It is the ideal solution for real-time processing use cases like streaming ETL or RAG pipelines for unstructured data.\n",
    "\n",
    "This Jupyter notebook is an introduction to Pathway.\n",
    "You will learn how to develop a simple pipeline over a data stream.\n",
    "\n",
    "## Installing Pathway with Python 3.10+\n",
    "\n",
    "First, you need to install Pathway. You can do it with a simple pip command.\n",
    "Using the cell below, you install Pathway into a Python 3.10+ Linux runtime.\n",
    "\n",
    "> **If you are running in Google Colab, please run the colab notebook (Ctrl+F9)**, disregarding the 'not authored by Google' warning.\n",
    ">\n",
    "> **The installation and loading time is less than 1 minute**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "pip-installation-pathway",
   "metadata": {
    "cellView": "form",
    "id": "pip-installation-pathway"
   },
   "outputs": [],
   "source": [
    "%%capture --no-display\n",
    "!pip install pathway"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d521d02c",
   "metadata": {
    "id": "d521d02c",
    "jp-MarkdownHeadingCollapsed": true,
    "lines_to_next_cell": 0
   },
   "source": [
    "## A moving average with Pathway\n",
    "In this Jupyter notebook, you will learn how to connect to a data source, design a data pipeline, and run it over both static and streaming data.\n",
    "\n",
    "This notebook shows you how to compute an moving average over a data stream.\n",
    "You will use Pathway, [Bokeh](https://bokeh.org/) and [Panel](https://panel.holoviz.org/) to build a real-time data visualisation.\n",
    "You can run the code for yourself [in Colab]() or [from Github]().\n",
    "\n",
    "Because Pathway [unites static and stream processing](https://pathway.com/developers/user-guide/connecting-to-data/switch-from-batch-to-streaming/#switching-from-batch-to-streaming) in a single syntax, you can use the exact same code for both batch and streaming. This way you can test your workflow with a static snapshot and then simply switch out the data source for a live stream when you want to visualise the full data flow in real-time.\n",
    "\n",
    "Let's jump in! 🪂"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f829fbc1",
   "metadata": {
    "id": "f829fbc1"
   },
   "source": [
    "## Imports and setup\n",
    "\n",
    "First, import the necessary libraries:\n",
    "\n",
    "- [`pathway`](/developers/user-guide/introduction/welcome/) for data processing,\n",
    "- `datetime` for date manipulation,\n",
    "- `panel` and `bokeh` for dashboarding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aee376fc",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 17
    },
    "id": "aee376fc",
    "outputId": "54d8d3c9-112b-46e9-84df-9f0ab121e15b"
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "import bokeh.plotting\n",
    "import panel\n",
    "\n",
    "import pathway as pw\n",
    "\n",
    "import logging\n",
    "logging.basicConfig(level=logging.CRITICAL, force=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6b0f9a3",
   "metadata": {
    "id": "a6b0f9a3"
   },
   "source": [
    "And then fetch the sample data from Github. This data was generated using [polygon.io](https://polygon.io) and replicate a financial data stream."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5ad4e82",
   "metadata": {
    "id": "d5ad4e82"
   },
   "outputs": [],
   "source": [
    "%%capture --no-display\n",
    "!wget -nc https://gist.githubusercontent.com/janchorowski/e351af72ecd8d206a34763a428826ab7/raw/ticker.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f3725d5",
   "metadata": {
    "id": "8f3725d5"
   },
   "source": [
    "## Data source setup\n",
    "\n",
    "In Pathway, the data is accessed using [connectors](https://pathway.com/developers/user-guide/connecting-to-data/connectors).\n",
    "You need to specify the schema of the [schema](https://pathway.com/developers/user-guide/types-in-pathway/schema) of the table.\n",
    "Since the data is stored in a CSV file, you can [automatically extract the schema](https://pathway.com/developers/user-guide/types-in-pathway/schema-generation):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "KlgbFoSJI22q",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KlgbFoSJI22q",
    "outputId": "424fec8b-e4ec-44cd-8a04-47706eb1d0ce"
   },
   "outputs": [],
   "source": [
    "fname = \"ticker.csv\"\n",
    "schema = pw.schema_from_csv(fname)\n",
    "print(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aDowhn2fRwOS",
   "metadata": {
    "id": "aDowhn2fRwOS"
   },
   "source": [
    "\n",
    "First, you are going to use a static sample of the data to develop and test our moving average.\n",
    "Let's extract the ten first lines of the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5Qvi-jqNS1Ix",
   "metadata": {
    "id": "5Qvi-jqNS1Ix"
   },
   "outputs": [],
   "source": [
    "!head -n 10 ticker.csv > ticker_head.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "KghDZu7sS2qD",
   "metadata": {
    "id": "KghDZu7sS2qD"
   },
   "source": [
    "You can now load the data using the [CSV connector](https://pathway.com/developers/user-guide/connectors/csv_connectors) with the mode set to \"static\" as you want to access static data and not a data stream:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "GejLv7bcJA4E",
   "metadata": {
    "id": "GejLv7bcJA4E"
   },
   "outputs": [],
   "source": [
    "static_data = pw.io.csv.read(\"ticker_head.csv\", schema=schema, mode=\"static\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9gQ7jtqjTY5E",
   "metadata": {
    "id": "9gQ7jtqjTY5E"
   },
   "source": [
    "You can print the data using [`pw.debug.compute_and_print`](https://pathway.com/developers/api-docs/debug#pathway.debug.compute_and_print).\n",
    "Note that this function only works in the static mode and should only be used in a debugging scenario. To serve your data out of Pathway, you should use an [output connector](https://pathway.com/developers/user-guide/introduction/concepts#send-the-results-to-external-systems-using-output-connectors)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "606gEaNLJLzQ",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "606gEaNLJLzQ",
    "outputId": "cd300f71-8645-4d14-92aa-ee2827754433"
   },
   "outputs": [],
   "source": [
    "pw.debug.compute_and_print(static_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0hmGmUDwUL6n",
   "metadata": {
    "id": "0hmGmUDwUL6n"
   },
   "source": [
    "Perfect, our data sample is read!\n",
    "\n",
    "Now, let's define a moving average of one minute.\n",
    "You will compute the 1-minute running mean of Volume Weighted Average Price (vwap) using the volume of transactions. You don't need to fully understand the financial terminology. All you need to know is that you are going to compute the mean of the price over a minute. The column `vwap` is the average price computed over the number of transactions contained in the column `volume`.\n",
    "\n",
    "First, you need to convert the time, contained in the column `t` as a timestamp, to a UTC format.\n",
    "Then, you can easily do a moving average using a [tumbling window](https://pathway.com/developers/user-guide/temporal-data/windows-manual#temporal-tumbling-windowing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "negfpAmDJ2ol",
   "metadata": {
    "id": "negfpAmDJ2ol"
   },
   "outputs": [],
   "source": [
    "def pipeline(input_data):\n",
    "  # Parse the timestamps\n",
    "  data = input_data.with_columns(t=input_data.t.dt.utc_from_timestamp(unit=\"ms\"))\n",
    "  # Compute the one minute average\n",
    "  minute_1_stats = (\n",
    "    data.windowby(\n",
    "        pw.this.t,\n",
    "        window=pw.temporal.tumbling(datetime.timedelta(minutes=1)),\n",
    "        behavior=pw.temporal.exactly_once_behavior(),\n",
    "        instance=pw.this.ticker,\n",
    "    )\n",
    "    .reduce(\n",
    "        ticker=pw.this._pw_instance,\n",
    "        t=pw.this._pw_window_end,\n",
    "        volume=pw.reducers.sum(pw.this.volume),\n",
    "        transact_total=pw.reducers.sum(pw.this.volume * pw.this.vwap),\n",
    "    )\n",
    "    .with_columns(vwap=pw.this.transact_total / pw.this.volume)\n",
    "  )\n",
    "  return minute_1_stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "PGlQkXvfKRN5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "PGlQkXvfKRN5",
    "outputId": "6f24e9e1-b9a3-45dd-8e5b-243e5221c1fa"
   },
   "outputs": [],
   "source": [
    "static_average = pipeline(static_data)\n",
    "pw.debug.compute_and_print(static_average)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9736cd85",
   "metadata": {
    "id": "9736cd85"
   },
   "source": [
    "## Switching between static and stream processing\n",
    "\n",
    "Thanks to the static data sample, you have developed and tested a data pipeline.\n",
    "Now, let's run it on live data!\n",
    "\n",
    "For simplicity, you will create a streaming data source that replays data in a CSV file using the [demo module](https://pathway.com/developers/user-guide/connecting-to-data/artificial-streams/). This is an easy way to simulate a live data stream without any infrastructure hassle. You can of course use Pathway with a real, production-grade data stream, for example [from Kafka](https://pathway.com/developers/user-guide/exploring-pathway/from-jupyter-to-deploy/#kafka-integration) or [Redpanda](https://pathway.com/developers/tutorials/connectors/switching-to-redpanda/).\n",
    "\n",
    "The `input_rate` parameter controls how fast the data is replayed.\n",
    "\n",
    "Then you can apply the same pipeline to the new data source.\n",
    "This is the only change you need to make in your code to switch between static and stream processing. Learn more about Pathway's unified engine and single syntax in [our User Guide](https://pathway.com/developers/user-guide/getting-started/switch-from-batch-to-streaming#switching-from-batch-to-streaming).\n",
    "\n",
    "\n",
    "💡 No data processing actually happens when you run the cell below. We are building a computational graph that will only be executed at the end of the notebook. This allows Pathway to optimise the computations and perform them as fast as possible when the data starts streaming."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce1b4ebd",
   "metadata": {
    "id": "ce1b4ebd"
   },
   "outputs": [],
   "source": [
    "live_data = pw.demo.replay_csv(fname, schema=schema, input_rate=1000)\n",
    "\n",
    "streaming_average = pipeline(live_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c03b3af2",
   "metadata": {
    "id": "c03b3af2",
    "lines_to_next_cell": 2
   },
   "source": [
    "## Dashboard creation\n",
    "\n",
    "Now create a `Bokeh` plot and `Panel` table visualization: the plot shows the Bollinger Bands along with the price running mean and indicates the price of buy and sell decisions. The table gathers all the decisions conveniently for further processing, such as reducing it to compute a historical evaluation of the gains of the strategy.\n",
    "\n",
    "When the cell is executed, placeholder containers are created for the plot and table visualization. They will be populated with live data when the computation is started (i.e. when running the final `pw.run()` at the end of this notebook)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8dd3d575",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 472
    },
    "id": "8dd3d575",
    "outputId": "72c8c306-e816-407b-c4fc-2d8aa47a0068"
   },
   "outputs": [],
   "source": [
    "def stats_plotter(src):\n",
    "\n",
    "    fig = bokeh.plotting.figure(\n",
    "        height=400,\n",
    "        width=600,\n",
    "        title=\"1-minute moving average\",\n",
    "        x_axis_type=\"datetime\",\n",
    "        y_range=(188.5, 191),\n",
    "    )\n",
    "    fig.line(\"t\", \"vwap\", source=src)\n",
    "    return fig\n",
    "\n",
    "\n",
    "viz = panel.Row(\n",
    "    streaming_average.plot(stats_plotter, sorting_col=\"t\"),\n",
    ")\n",
    "viz"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27d8c7bc",
   "metadata": {
    "id": "27d8c7bc",
    "lines_to_next_cell": 0
   },
   "source": [
    "## Running the computation\n",
    "\n",
    "All the hard work is done! The final step is to start the Pathway data processing engine using the `pw.run()` command.\n",
    "**Do not scroll down once the cell is executed, scroll up to see the animation.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c24c9a05",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35,
     "referenced_widgets": [
      "1ceff99562914eeba712dc5fb2c3399f",
      "cb52e6396ed047ebbde6784e6aa65aa2"
     ]
    },
    "id": "c24c9a05",
    "lines_to_next_cell": 0,
    "outputId": "da92dff1-ac91-4b2d-9c3a-5b1600c7c31a"
   },
   "outputs": [],
   "source": [
    "pw.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e71febc-94b1-46f9-beff-aa167152122a",
   "metadata": {},
   "source": [
    "Watch how the dashboard is updated in realtime!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61b8fd80",
   "metadata": {
    "id": "61b8fd80"
   },
   "source": [
    "While the computation is running, `pathway` prints important statistics such as message processing latency.\n",
    "\n",
    "You can do more advanced computation such as the [Bollinger Bands](https://pathway.com/developers/showcases/live_data_jupyter)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a3e220b",
   "metadata": {
    "id": "8a3e220b"
   },
   "source": [
    "## What else can you do with Pathway?\n",
    "\n",
    "Congratulations! You have developed your first streaming pipeline with Pathway. Easy right?\n",
    "\n",
    "* Perform machine learning in real time. e.g. [ real-time\n",
    "Classification](/developers/showcases/lsh/lsh_chapter1/) , [real-time fuzzy joins](/developers/showcases/fuzzy_join/fuzzy_join_chapter2/)\n",
    "\n",
    "* Transform unstructured data to structured data using [live LLM pipelines](https://github.com/pathwaycom/llm-app)\n",
    "\n",
    "* Making [joins](/developers/tutorials/fleet_eta_interval_join/) simple with timeseries data\n",
    "\n",
    "And so much more... Read more about what we can do in the [developer docs](/developers/user-guide/introduction/welcome/).\n",
    "\n",
    "We would love to have you try out [Pathway on GitHub](https://github.com/pathwaycom/pathway)."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "1ceff99562914eeba712dc5fb2c3399f": {
     "model_module": "@jupyter-widgets/output",
     "model_module_version": "1.0.0",
     "model_name": "OutputModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/output",
      "_model_module_version": "1.0.0",
      "_model_name": "OutputModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/output",
      "_view_module_version": "1.0.0",
      "_view_name": "OutputView",
      "layout": "IPY_MODEL_cb52e6396ed047ebbde6784e6aa65aa2",
      "msg_id": "",
      "outputs": [
       {
        "data": {
         "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">                                            PATHWAY PROGRESS DASHBOARD                                             \n                                                                                                                   \n   <span style=\"font-weight: bold\">             </span> <span style=\"font-weight: bold\">        no. </span> <span style=\"font-weight: bold\">             </span> <span style=\"font-weight: bold\">            </span>  <span style=\"font-weight: bold\">          </span> <span style=\"font-weight: bold\"> latency to wall clock </span> <span style=\"font-weight: bold\">                   </span>   \n   <span style=\"font-weight: bold\">             </span> <span style=\"font-weight: bold\">   messages </span> <span style=\"font-weight: bold\">             </span> <span style=\"font-weight: bold\">            </span>  <span style=\"font-weight: bold\"> operator </span> <span style=\"font-weight: bold\">                  [ms] </span> <span style=\"font-weight: bold\"> lag to input [ms] </span>   \n   <span style=\"font-weight: bold\">             </span> <span style=\"font-weight: bold\">     in the </span> <span style=\"font-weight: bold\">             </span> <span style=\"font-weight: bold\">            </span>  ──────────────────────────────────────────────────────   \n   <span style=\"font-weight: bold\">             </span> <span style=\"font-weight: bold\">       last </span> <span style=\"font-weight: bold\"> in the last </span> <span style=\"font-weight: bold\">      since </span>   input                   finished                        \n   <span style=\"font-weight: bold\"> connector   </span> <span style=\"font-weight: bold\">  minibatch </span> <span style=\"font-weight: bold\">      minute </span> <span style=\"font-weight: bold\">      start </span>   output                         0                None    \n   ─────────────────────────────────────────────────────                                                           \n    PythonRead…     finished         18316        18316  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; font-style: italic\">   Above you can see the latency of input and output    </span>  \n                                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; font-style: italic\">  operators. The latency is measured as the difference  </span>  \n                                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; font-style: italic\"> between the time when the operator processed the data  </span>  \n                                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; font-style: italic\">      and the time when pathway acquired the data.      </span>  \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                       LOGS                                                        \n  <span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[03/20/24 14:59:27] </span><span style=\"color: #800000; text-decoration-color: #800000\">WARNING </span> PythonReader-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>: Closing the data source                                             \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n</pre>\n",
         "text/plain": "                                            PATHWAY PROGRESS DASHBOARD                                             \n                                                                                                                   \n   \u001b[1m             \u001b[0m \u001b[1m \u001b[0m\u001b[1m       no.\u001b[0m\u001b[1m \u001b[0m \u001b[1m             \u001b[0m \u001b[1m            \u001b[0m  \u001b[1m          \u001b[0m \u001b[1m \u001b[0m\u001b[1mlatency to wall clock\u001b[0m\u001b[1m \u001b[0m \u001b[1m                   \u001b[0m   \n   \u001b[1m             \u001b[0m \u001b[1m \u001b[0m\u001b[1m  messages\u001b[0m\u001b[1m \u001b[0m \u001b[1m             \u001b[0m \u001b[1m            \u001b[0m  \u001b[1m \u001b[0m\u001b[1moperator\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m                 [ms]\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1mlag to input [ms]\u001b[0m\u001b[1m \u001b[0m   \n   \u001b[1m             \u001b[0m \u001b[1m \u001b[0m\u001b[1m    in the\u001b[0m\u001b[1m \u001b[0m \u001b[1m             \u001b[0m \u001b[1m            \u001b[0m  ──────────────────────────────────────────────────────   \n   \u001b[1m             \u001b[0m \u001b[1m \u001b[0m\u001b[1m      last\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1min the last\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     since\u001b[0m\u001b[1m \u001b[0m   input                   finished                        \n   \u001b[1m \u001b[0m\u001b[1mconnector  \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m minibatch\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     minute\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     start\u001b[0m\u001b[1m \u001b[0m   output                         0                None    \n   ─────────────────────────────────────────────────────                                                           \n    PythonRead…     finished         18316        18316  \u001b[2;3m   Above you can see the latency of input and output    \u001b[0m  \n                                                         \u001b[2;3m  operators. The latency is measured as the difference  \u001b[0m  \n                                                         \u001b[2;3m between the time when the operator processed the data  \u001b[0m  \n                                                         \u001b[2;3m      and the time when pathway acquired the data.      \u001b[0m  \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                       LOGS                                                        \n  \u001b[2;36m[03/20/24 14:59:27]\u001b[0m\u001b[2;36m \u001b[0m\u001b[31mWARNING \u001b[0m PythonReader-\u001b[1;36m0\u001b[0m: Closing the data source                                             \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n"
        },
        "metadata": {},
        "output_type": "display_data"
       }
      ],
      "tabbable": null,
      "tooltip": null
     }
    },
    "cb52e6396ed047ebbde6784e6aa65aa2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "2.0.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "2.0.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "2.0.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border_bottom": null,
      "border_left": null,
      "border_right": null,
      "border_top": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
